Work in the 1990s developed the concept of using neural networks to solve partial differential equations (PDEs). Supported by recent advances in computational tools, Raissi et al. revisited this concept in 2019, proposing the physics-informed neural networks (PINNs) name and framework. PINNs minimize a loss function which includes the PDE residual determined for a set of collocation points. The number and distribution of these collocation points have a significant influence on the performance of the PINN. Adaptive collocation point sampling methods have been proposed before, such as residual-based adaptive distribution presented by Wu et al. in 2022. However, these adaptive sampling methods scale poorly to higher dimensions. In this work, we present the Point Marching Adaptive Collocation Method for PINNs which uses the gradient of the PDE residual as guiding information. Inspired by classic optimization problems, this method incrementally moves collocation points towards regions of higher residuals using established optimization algorithms such as gradient ascent or Adam. We test the effectiveness of the Point Marching Adaptive Collocation Method for three forward problems, including a high-dimensional problem, and a high-dimensional inverse problem. The numerical results demonstrate that this method with the Adam optimizer outperforms previous approaches in accuracy and computational efficiency, for the problems under consideration. Furthermore, due to the nature of this method, it efficiently scales to high-dimensional problems.